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Summary 


Many state education leaders are interested in finding schools that have demonstrated 
success in improving the achievement of students at the highest risk for difficulties. These 
schools are typically identified by comparing observed performance on an exam, such as a 
state assessment exam, with expected performance based on demographic characteristics, 
including the percentage of students classified as economically disadvantaged (proxied by 
eligibility for free or reduced-price lunch), as a racial/ethnic minority, or as an English 
language learner. 

This study used data from the Florida Department of Education on public elementary 
schools for 2012/13 to demonstrate methods for answering two general research questions: 

• Which schools are exceeding student achievement expectations, given the demo- 
graphic characteristics of their students? 

• What demographic similarities exist between schools that are exceeding expecta- 
tions and other schools? 

Using a multiple regression analysis, the study demonstrated that of Florida’s roughly 
2,000 public elementary schools, 43 (about 2 percent) are exceeding expectations in 
grade 3 reading (schools “beating the odds”). These schools had between 14 percent and 
29 percent fewer students scoring at the lowest achievement level (level 1) on the statewide 
assessment than would be predicted when controlling for the demographic characteristics 
of their students. And in 2012/13 the average difference between a school’s observed and 
expected percentage of students scoring at achievement level 1 was about ±7 percent, with 
a range of 29 percent fewer students observed than expected to 73 percent more students 
observed than expected. 

Profile analysis was used to illustrate that the number of schools beating the odds in 
Florida varied by school demographic profile, with most having above-average percentages 
of Black students and economically disadvantaged students. 
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Why this study? 


Florida law requires that grade 3 students scoring at the lowest achievement level (level 1) 
on the Florida Comprehensive Assessment Test 2.0 (FCAT 2.0) in reading be retained, 
unless they can demonstrate the required reading level on an alternative nationally stan- 
dardized reading assessment or through a portfolio assessment (Florida Department of 
Education, 2013). This requirement has made grade 3 reading performance highly visible, 
particularly for students who score at achievement level 1. The Regional Educational Lab' 
oratory Southeast Improving Literacy Alliance requested an analysis of the results of the 
2013 grade 3 FCAT 2.0 reading assessment to identify schools with a lower-than-expected 
percentage of students scoring at achievement level 1 (schools “beating the odds”), with 
the intention of studying these schools to understand what practices might be able to 
further support grade 3 reading instruction. 

This study was designed to address the specific request made by the Improving Litera- 
cy Alliance, and more broadly, demonstrate methods for answering two general research 
questions: 

• Which schools are exceeding student achievement expectations, given the demo- 
graphic characteristics of their students? 

• What demographic similarities exist between schools that are exceeding expecta- 
tions and other schools? 

See box 1 for a description of the study data. See appendix A for details on the analyses. 

Study findings 


About 2 percent of 
the roughly 2,000 
Florida public 
elementary schools 
had a negative 
difference between 
the observed 
and predicted 
percentage of their 
students scoring 
at achievement 
level 1 that was 
large enough to be 
considered reliable 


This section describes the results of the regression analyses used to identify schools exceed- 
ing student achievement expectations, as well as the profile analysis used to describe the 
demographic characteristics of those schools. 


About 2 percent of schools beat the odds for the expected percentage of students scoring at achievement 
level 1 on the grade 3 reading assessment, given the demographic characteristics of their students 

Schools beating the odds can be identified by comparing how their students perform on 
an assessment with how those students would be expected to perform when accounting for 
their demographic characteristics. Negative differences indicate that a school had a low- 
er-than-expected percentage of students scoring at achievement level 1; positive differences 
indicate a higher-than-expected percentage. 

In 2012/13 the average difference between a school’s observed and expected percentage 
of students scoring at achievement level 1 was about 7 percent. The range was 29 percent 
fewer students observed than expected to 73 percent more students observed than expect- 
ed, with a standard error of ±7.27 percent. 

In 2012/13, 1,096 of the roughly 2,000 Florida public elementary schools had a negative 
difference between the observed and predicted percentage of their students scoring at 
achievement level 1. Of these 1,096, 43 (about 2 percent of public elementary schools state- 
wide) had a difference that was large enough to be considered reliable. (See appendix A 
for a discussion of the 95 percent confidence interval.) These 43 schools beat the odds in 
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Box 1. Study data 

The Florida Department of Education provided data on each public elementary school for 
2010/11-2012/13, 1 including: 

• Number of students tested. 

• Number of students scoring at achievement level 1 on the grade 3 Florida Comprehensive 
Assessment Test 2.0 (FCAT 2.0) in reading (outcome measure). 

• Number of students classified as White, Black, and FHispanic. 

• Number of students classified as eligible for free or reduced-price lunch (measure of eco- 
nomic disadvantage). 

• Number of students classified as English language learner students. 

• School number and name. 

• District number and name. 

State-level data on race/ethnicity, eligibility for free or reduced-price lunch, and English 
language learner status were downloaded from the Florida Department of Education’s demo- 
graphic database. The data show that Black students and English language learner students 
are over-represented among students scoring at achievement level 1 given their proportion of 
the state sample (see table). 


Box table. Percentage of grade 3 students scoring at achievement level 1 on the Florida 
state reading assessment, by race/ethnicity, English language learner status, and eligibility 
for free or reduced-price lunch, 2012/13 


Group 

Number of 
students 

Percentage of 
state sample 

Percentage scoring 
at achievement 
level 1 

White 

82,265 

40 

10 

Black 

47,101 

23 

29 

Hispanic 

63,113 

31 

21 

English language learner 

24,011 

12 

46 

Eligible for free or reduced-price lunch 

133,004 

65 

24 

Total 

206,006 

na 

18 


na is not applicable. 

Source: Authors' analysis of data obtained from the Florida Department of Education's demographic database. 

Comparisons were made using the grade 3 FCAT 2.0 reading test results, and the differ- 
ence between the observed and expected percentages was the measure of a school’s success 
in exceeding expectations with high-risk readers in grade 3. Negative differences indicate that 
a school had a lower-than-expected percentage of students scoring at achievement level 1; 
positive differences indicate a higher-than-expected percentage. 

1. While three years of data were available and modeled, the results are based on data for 2012/13 due to 
stable results across years. See appendix A for information on the analyses. 


grade 3 reading during the 2012/13 school year, with 14-29 percent fewer students scoring at 
achievement level 1 than would be predicted when controlling for the demographic charac- 
teristics of their students. Appendix B lists statistics for the 43 schools. 

The number of schools beating the odds varied by school demographic profile. Most 
schools beating the odds had above-average percentages of Black students and economi- 
cally disadvantaged students. 
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About 70 percent of the schools beating the odds had higher percentages of Black and 
economically disadvantaged students than the average for Florida schools 


To determine what demographic similarities exist between schools beating the odds and 
other schools in Florida, the study used latent profile analysis (see appendix A) to identify 
school demographic profiles for all Florida elementary schools based on grade 3 students. 


The results yielded five distinct profiles (table 1). Each profile was named for its predonv 
inant student demographic characteristic or characteristics, with the term “high” used 
when the percentage of students in the category was above the state average and exceeded 
70 percent. 

• Profile 1 . High percentage of Hispanic students and high percentage of econom- 
ically disadvantaged students (11 percent of schools). Schools fitting this profile 
also had a higher percentage of English language learner students than the state 
average. 

• Profile 2. General population (29 percent of schools). Schools fitting this profile 
reflect the state averages. 

• Profile 3. High percentage of Black students and high percentage of economically 
disadvantaged students (17 percent of schools). 

• Profile 4 • High percentage of White students (30 percent of schools). 

• Profile 5. High percentage of economically disadvantaged students (13 percent of 
schools). Schools fitting this profile also had higher percentages of Hispanic and 
English language learner students than the state average. 

Of the 43 schools identified as beating the odds, profile 3 accounted for the highest per- 
centage, at 70 percent, followed by profile 2 at 14 percent, profile 1 at 9 percent, and 
profile 5 at 7 percent (figure 1). None of the schools in profile 4 beat the odds. 


Of the 43 schools 
identified as 
beating the odds, 
profile 3 (high 
percentage of 
Black students and 
high percentage 
of economically 
disadvantaged 
students) 
accounted for 
the highest 
percentage, at 
70 percent 


Table 1. Five demographic profiles of Florida public elementary schools, 2012/13 

Mean percentages, unless otherwise noted 


Demographic 

characteristic 

1. High Hispanic 
and high 
economically 
disadvantaged 

2. General 
population 

3. High Black 
and high 
economically 
disadvantaged 

4. High 
White 

5. High 
economically 
disadvantaged 

White 

6 

45 

6 

73 

21 

Black 

7 

21 

77 

8 

23 

Hispanic 

84 

25 

13 

12 

50 

English language 






learner 

32 

7 

10 

2 

18 

Eligible for free or 






reduced-price lunch 

83 

62 

91 

47 

77 

Number and percentage of schools 

Number 

218 

598 

352 

621 

270 

Percent 

11 

29 

17 

30 

13 


Source: Authors’ analysis of data obtained by special request from the Florida Department of Education. 
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Figure 1. Most schools beating the odds for the expected percentage of students 
scoring at achievement level 1 on the grade 3 Florida Comprehensive Assessment 
Test 2.0 in reading in 2012/13 fit demographic profile 3 



Note: Demographic profile 3 includes schools with percentages of Black students and students eligible for 
free or reduced-price lunch above the state average and exceeding 70 percent (see table 1). 

Source: Authors' analysis of data obtained by special request from the Florida Department of Education. 


Appendix B provides information on the 43 schools identified as beating the odds, includ- 
ing demographic characteristics, residual estimate, 95 percent confidence interval for the 
residual, and demographic profile number. 


Pairing schools 
with similar 
demographics 
based on their 
profile numbers 
would allow 
successful 
schools to mentor 
struggling schools 
with similar 
demographic 
characteristics 


Implications, limitations, and next steps 


The methods used in this study can be replicated in different contexts to identify schools 
exceeding expectations. Schools that are beating the odds can be studied for promising 
schooHevel practices. The profile numbers can be used to match schools with similar 
demographics, in a “nearest neighbor” approach. For example, in Florida a school match- 
ing profile 3 with a greater-than-expected percentage of students scoring at achievement 
level 1 might be paired with a school in profile 3 that has been identified as beating the 
odds. Such a pairing would allow successful schools to mentor struggling schools with 
similar demographic characteristics. 

Prior to applying these methods in other contexts, several study limitations should be con- 
sidered. First, the analyses were conducted using school-level data instead of student-level 
data. Using student-level data would allow for a more rigorous evaluation. In addition, the 
analyses include results for grade 3 only. Expanding the analyses to include data for all 
tested grades would allow for a more comprehensive evaluation of each school’s perfor- 
mance. Also, the analyses do not control for prior academic performance, which could 
identify advantaged or disadvantaged schools by the enrollment of higher or lower per- 
forming students in a given year. Analyses addressing these limitations might yield results 
different from those in this report. However, in Florida there are no statewide summative 
assessments of reading comprehension in grade 2 that could control for prior achievement. 

A final consideration is that the beating the odds estimates and ranking of schools 
depend on the schools included in the analysis, many of which serve a specialized student 
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population based on specified enrollment criteria (for example, a charter school serving 
high-performing students or a school whose population consists primarily of students with 
disabilities). Such schools have historically demonstrated higher or lower performance on 
statewide assessments. As a result, the inclusion of those schools may affect which schools 
are identified as beating the odds. Future work should include a sensitivity analysis to evah 
uate how the beating the odds scores change as particular schools are excluded, as well as 
which schools may be differentially identified as beating the odds. 
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Appendix A. Details on the analyses and results 


This study entailed a two-stage analysis. In the first stage the proportion of grade 3 stu- 
dents scoring at achievement level 1 on the Florida Comprehensive Assessment Test 2.0 
in reading was the outcome measure in a multiple linear regression prediction model using 
school demographic characteristics as predictors, represented by: 

Y. = B 0 + B 1 (White) j +B 1 (Black) j + B } (Hispanic ). + B 4 (ELL). + B 5 ( FRL). + B 6 (Y ear) j + e, 

where the dependent variable Y. is the proportion of students scoring at achievement level 1 
for school j, controlling for the proportion of students classified as White, Black, Hispanic, 
English language learner (ELL), and eligible for free or reduced-price lunch (FRL); Year, is 
a dummy coded variable for the assessment year; and e. is the school-level residual. The 
school-level residual served as the measure of a school’s success in exceeding expectations 
with grade 3 students at high risk of reading difficulty. 

Model estimates 

Several models were estimated. Model 1, which included data for the 2010/11, 2011/12, and 
2012/13 school years, was estimated to judge the stability of the results over time. Model 
2 included data for 2012/13 only, so it did not require the addition of the dummy variable 
Year.. Before the analyses were run, mean differences across years for each predictor were 
tested and found to be nonsignificant; therefore, interaction terms were not included in 
model 1. There were no missing data for the variables of interest. 

Model l’s R 2 of 0.61, reflecting the strength of the relationship between the proportion of 
students scoring at achievement level 1 and the independent variables, was statistically 
significant at the 0.05 level ( p < .0001). All predictors were statistically significant, except 
Year.. Dropping Year, from model 1 was supported by an analysis of the reduction in Akaike 
information criteria by the best combination of predictors. This analysis showed the most 
parsimonious model to be one that included all predictors except Year.. That Year, was not 
significant indicates that there are no differences in the intercept across years, controlling 
for school demographic characteristics. 

Model 1 was revised and re-estimated, resulting in an R 2 of 0.61. The effects of the inde- 
pendent variables on the proportion of students scoring at achievement level 1 are summa- 
rized in table Al. 

Model 2’s R 2 of 0.61, reflecting a relationship as strong as that in model 1, was also found to 
be statistically significant at the 0.05 level (p < .0001). All predictors were found to be sig- 
nificant, which was supported by an analysis of the reduction in Akaike information cri- 
teria. No revisions to the model were necessary. The effects of the independent variables 
on the proportion of students scoring at achievement level 1 are summarized in table A2. 

The mean and standard deviation of the residuals of model 1 were compared with those of 
model 2. The nearly identical results (table A3), as well as a statistically significant correla- 
tion coeffient (r = 0.998) between model residuals, demonstrated model stability over time. 
Given this stability, model 2 was selected for use in subsequent analyses. 
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Table Al. Model 1 results summary 

1 Variable 

Effect estimate 

Standard error 

Intercept 

-0.188** 

0.018 

White 

0.195** 

0.021 

Black 

0.319** 

0.020 

Flispanic 

0.139** 

0.020 

English language learner 

0.247** 

0.012 

Eligible for free or reduced-price lunch 

0.206** 

0.005 


** Significant at the .05 level. 
n = 6,099. 

Source: Authors' analysis of data obtained by special request from the Florida Department of Education. 


Table A2. Model 2 results summary 

1 Variable 

Effect estimate 

Standard error 9 

Intercept 

-0.208** 

0.031 

White 

0.222** 

0.035 

Black 

0.345** 

0.035 

Hispanic 

0.153** 

0.034 

English language learner 

0.298** 

0.021 

Eligible for free or reduced-price lunch 

0.195** 

0.009 


** Significant at the .05 level. 
n = 2,059. 

Source: Authors' analysis of data obtained by special request from the Florida Department of Education. 


Table A3. Model 1 and model 2 residuals 


Model Mean Standard deviation 


Model 1 residuals (pooled sample) -0.001 0.073 

Model 2 residuals 0.000 0.073 


Source: Authors' analysis of data obtained by special request from the Florida Department of Education. 


The model 2 residuals were used to evaluate which schools’ observed estimates were greater 
than expected (they performed worse than expected by having a greater-than-expected 
percentage of students scoring at achievement level 1) and which schools’ observed esti- 
mates were lower than expected (they performed better than expected by having a lower- 
than-expected percentage of students scoring at achievement level 1). 

Because residuals have a mean of zero, the range of residuals was important for contextual- 
izing the differences between the observed and predicted percentages for the full sample of 
schools. Additionally, the standard error of the residuals — a measure of their precision — 
provides information for identifying which schools could reliably be identified as beating 
the odds given the data. The range of the residuals was -29 percent to 73 percent, with a 
standard error of ±7.27 percent. 
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A 95 percent confidence interval was constructed around each school’s residual, using the 
formula: 

95 percent confidence interval = School residual ±1.96 x Residual standard error. 

This confidence interval allowed for the determination of whether a value of zero was 
just as plausible as the estimated residual. If zero is contained within a school’s 95 percent 
confidence interval, whether a school is beating the odds cannot be determined reliably. 
Schools were identified as beating the odds if the 95 percent confidence interval did not 
include zero. Of the 1,096 schools with a negative residual, 43 had a negative residual with a 
95 percent confidence interval that did not include zero (a residual less than -14.3 percent). 
The residuals of these schools ranged from -28.76 percent to -14.34 percent. 

To illustrate, the school with the largest negative residual was predicted to have 29 percent 
of its students scoring at achievement level 1. The observed percentage in 2013 was 
0 percent, resulting in a residual of -29 percent. The 95 percent confidence interval for 
this school was -14 percent to -43 percent. The 95 percent confidence interval did not 
include zero, indicating a significant difference between the observed and predicted scores. 
Using this approach, it can be estimated that any school with a negative residual greater 
than -0.143 would no longer be considered as having beat the odds because its 95 percent 
confidence interval would include zero. 

Latent profile analysis of school demographic characteristics 

The second stage of the analysis featured a latent profile analysis of school demographic 
characteristics. This clustering of schools provides a “nearest neighbor” comparison, allow- 
ing the Florida Department of Education to compare the residual scores from the first stage 
with those that are most similar based on demographic characteristics. 

Latent profile analysis is typically used to classify individuals into groups based on their 
responses on a single exam or on their scores on multiple exams. And though it has been 
used predominantly for diagnostic purposes in psychology, it is an emerging descriptive 
classification technique in education (Logan & Petscher, 2010). While latent profile anal- 
ysis is descriptive, its utility lies in an ability to empirically categorize participants (in this 
case, schools) into similar groups based on variables of interest (in this case, school demo- 
graphic characteristics). For example, if the latent profile analysis was used to identify two 
profiles (or groups) of schools, there would most likely be one group of schools with high 
percentages of students with high risk of reading difficulty and one group with low per- 
centages. The schools with high percentages would be considered high-risk, while those 
with low percentages would be considered low-risk. Then, within a profile, the variation in 
individual school outcomes can be described (for example, school residual scores). 

The equation below shows the basic representation of a multivariate latent profile analysis 
model (Pastor, Barron, Miller, & Davis, 2007): 

/W = 2 fc _ t 

where y. represents the multivariate distribution of cluster indicators (school demograph- 
ics) for school i (with the number of clusters represented by k), 0 represents the unique set 


A-3 


of model parameters to be estimated within each cluster, and n k is the weight given to each 
cluster. The weights are constrained to be non-negative and must sum to 1. Each cluster 
distribution is defined by u k (the mean vector) and (the covariance matrix). 

Multiple indices were used to determine which number of profiles was the most appropriate 
for the data (table A4). The indices include Akaike information criteria (Kaplan, 2000), 
Bayesian information criteria (Kaplan, 2000), entropy (Ramaswamy, DeSarbo, Reibstein, 
& Robinson, 1993), and two tests reported in the MPlus program (Muthen & Muthen, 
1998-2012) — the Lo-Mendell-Rubin likelihood ratio test (Lo, Mendell, & Rubin, 2001) 
and a parametric bootstrapped likelihood ratio test (McLachlan & Peel, 2000). 

A five-class model was selected as the best fit to the data. Moving to six classes result- 
ed in a nonsignificant p-value for the Lo-Mendell-Rubin likelihood ratio test. In addition 
to fitting the data well, the five-class model resulted in a solution that lent itself well to 
interpretation. 


Table A4. Summary of latent profile analysis model fit indices 


Number of 
classes 

Degrees of 
freedom 

Akaike 

information 

criteria 

Bayesian 

information 

criteria 

Adjusted 

Bayesian 

information 

criteria 

Entropy 

Adjusted 

LMR 

LMR 

(p-value) 

BLRT 

(p-value) 

1 

10 

-1,444.36 

-1,388.07 

-1,419.84 





2 

16 

-4,629.27 

-4,539.19 

-4,590.02 

0.88 

3,128.56 

.0000 

.0000 

3 

22 

-8,221.30 

-8,097.47 

-8,167.37 

0.95 

3,930.02 

.0000 

.0000 

4 

28 

-9,807.50 

-9,649.87 

-9,738.82 

0.92 

1,564.01 

.0000 

.0000 

5 

34 

-10,524.19 

-10,332.77 

-10,440.79 

0.91 

713.11 

.0001 

.0000 

6 

40 

-11,203.29 

-10,978.09 

-11,105.18 

0.91 

676.33 

.0737 

.0000 


LMR is Lo-Mendell-Rubin likelihood ratio test. BLRT is boot-strapped likelihood ratio test. 

Source: Authors' analysis of data obtained by special request from the Florida Department of Education. 
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Appendix B. Statistics for schools beating the odds 
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